Automated profiling of resource usage

ABSTRACT

Operating profiles for consumers of computing resources may be automatically determined based on an analysis of actual resource usage measurements and other operating metrics. Measurements may be taken while a consumer, such as a virtual machine instance, uses computing resources, such as those provided by a host. A profile may be dynamically determined based on those measurements. Profiles may be generalized such that groups of consumers with similar usage profiles are associated with a single profile. Assignment decisions may be made based on the profiles, and computing resources may be reallocated or oversubscribed if the profiles indicate that the consumers are unlikely to fully utilize the resources reserved for them. Oversubscribed resources may be monitored, and consumers may be transferred to different resource providers if contention for resources is too high.

BACKGROUND

Generally described, computing devices utilize a communication network,or a series of communication networks, to exchange data. Companies andorganizations operate computer networks that interconnect a number ofcomputing devices to support operations or provide services to thirdparties. The computing systems can be located in a single geographiclocation or located in multiple, distinct geographic locations (e.g.,interconnected via private or public communication networks).Specifically, data centers or data processing centers, herein generallyreferred to as “data centers,” may include a number of interconnectedcomputing systems to provide computing resources to users of the datacenter. The data centers may be private data centers operated on behalfof an organization or public data centers operated on behalf, or for thebenefit of, the general public.

To facilitate increased utilization of data center resources,virtualization technologies may allow a single physical host computingdevice to host one or more instances of virtual machine instanceconfigurations that appear and operate as independent computing devicesto users of a data center. With virtualization, the single physical hostcomputing device can create, maintain, delete, or otherwise managevirtual machine instances in a dynamic manner. In turn, users canrequest single computing devices or a configuration of networkedcomputing devices, and be provided with varying numbers of virtualmachine resources.

The computing resources provided by the host computing devices mayinclude computing capacity, memory and other storage, bandwidth, and thelike. In a data center environment with thousands of host computingdevices, an instance of a virtual machine may be instantiated on arandom host computing device so long as the target host computing devicemeets specified criteria such as sufficient and available computingdevice resources (e.g., processing units, memory, and the like). Once aninstance of a virtual machine is instantiated on a physical hostcomputing device, a predetermined amount of one or more computingresources may be reserved for use by the virtual machine instance. Acomputing resource provider or other operator of the data centerenvironment may guarantee availability, to the virtual machine instance,of the reserved amounts of computing resources on the target computingdevice.

BRIEF DESCRIPTION OF DRAWINGS

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

FIG. 1 is a block diagram of an illustrative network computingenvironment including a management component, multiple host computingdevices, and multiple customers.

FIG. 2 is a block diagram of an illustrative management componentincluding various modules and storage components.

FIG. 3 is a flow diagram of an illustrative process for automaticallyprofiling computing resource usage.

FIG. 4 is a block diagram of an illustrative host computing devicehosting virtual machines which utilize computing resources provided bythe computing device.

FIG. 5 is a flow diagram of an illustrative process for launchingvirtual machines on host computing devices, allocating andoversubscribing computing resources, and migrating currently executingvirtual machines in order to further optimize computing resourceutilization.

FIG. 6 is a block diagram of an illustrative host computing device inwhich various computing resources are oversubscribed and excess capacityremains available.

FIG. 7 is a block diagram of an illustrative host computing device inwhich various computing resources are substantially oversubscribed dueto minimal resource utilization of each virtual machine executing on thehost computing device.

FIG. 8 is a block diagram of an illustrative migration of a virtualmachine from one host computing device to another host computing devicedue to oversubscription and a lack of available computing resourcecapacity.

DETAILED DESCRIPTION

Generally described, the present disclosure relates to computingresource utilization. Specifically, the disclosure relates toautomatically determining resource usage and operating metric profilesfor consumers of computing resources based on an analysis of actualresource usage measurements and other operating metrics. In someembodiments, a consumer of computing resources, such as an instance of avirtual machine instantiated according to customer specifications (e.g.:a virtual machine instance instantiated from a virtual machine imageconfigured with an operating system and application software accordingto customer specifications), may be instantiated on a host physicalcomputing device. The virtual machine instance may consume variouscomputing resources based on the execution of one or more computersoftware programs or other workloads by the virtual machine instance.The virtual machine instance can then terminate execution or otherwisebe configured for a different purpose.

For specific entities, such as a customer or set of customers, theprocess of instantiating virtual machine instances may be repeated. Aspart of processing the lifecycle of the virtual machine instance, aservice provider associated with providing the virtual machine instancescan observe and record resource consumption. The service provider canthen determine a virtual machine instance resource usage and operatingmetric profile based on processing resource consumption measurements andother operating metric information.

Additional aspects of the disclosure relate to generalizing the resourceusage and operating metric profiles, generally referred to as operatingprofiles. For example, thousands or more of virtual machine instancesmay be instantiated and may utilize resources in a single networkcomputing environment, such as a data center. Rather than determiningand maintaining separate customized operating profiles for each virtualmachine instance, virtual machine instances may be assigned togeneralized or default operating profiles. Accordingly, groups ofvirtual machine instances may be categorized according to a variety oforganizational criteria and assigned to the same operating profile. Insome cases, the operating profiles may be hierarchical, such that aparticular virtual machine instance configuration is associated with aparticular operating profile, and also with a more general operatingprofile that is itself associated with multiple virtual machine instanceconfigurations. There may be multiple levels to the hierarchy, withpotentially thousands of virtual machine instance configurationsassociated with a single general operating profile or a small number oftop-level operating profiles, and a larger number of more specificoperating profiles at each level within the hierarchy. At the bottomlevel of the hierarchy may be a particular operating profile associatedwith a single virtual machine instance configuration as used by aparticular customer.

Further aspects of the disclosure relate to identifying a target hostcomputing device to provide computing resources to virtual machineinstances based on an automatically determined operating profile. Insome embodiments, particular physical host computing devices may beconfigured to provide target computing resources to multiple virtualmachine instances concurrently. A predetermined amount of a computingresource may be reserved for use by a single virtual machine instance.When the operating profile for a virtual machine instance indicates thatthe virtual machine instance will not likely consume computing deviceresources that have been reserved for the instance, the host computingdevice may instantiate additional virtual machine instances. Theadditional virtual machine instances may be associated with, or haveaccess to, host computing device resources concurrently with thepreviously instantiated virtual machine instances. Additionally, if anyof the virtual machine instances consumes or otherwise restricts accessto a resource such that the consumption meets or exceeds one or morethresholds specified in an operating profile for the virtual machineinstance, one or more of the virtual machine instances may betransferred to another host computing device. For example, the virtualmachine instance associated with consumption that exceeds a thresholdmay be transferred, or other virtual machine instances may betransferred.

Some computing resources are not necessarily provided by the hostcomputing devices, but rather are off-host resources. For example,network topology may require communications from one host computingdevice to a second host computing device to traverse one or more links(e.g., network connections between switches and other networkcomponents). The number of links may be different for communications toa third host computing device. Data regarding link traffic and thenumber of links that communications will traverse between hosts may berecorded as a resource consumption measurement or operating metric. Thedata may be used in the future to determine on which host computingdevice to instantiate a virtual machine instance that may communicatewith another host computing device. In addition, consumption of somecomputing resources does not necessarily reduce the amount of theresource that is available for other virtual machine instances orconsumers generally. For example, a feature provided by a host computingdevice, such as a particular instruction set, may be generally referredto as a computing resource. Usage of the instruction set, however, doesnot necessarily reduce availability of the instruction set to anothervirtual machine instance, application, or other consumer.

Although aspects of the embodiments described in the disclosure willfocus, for the purpose of illustration, on relationships andinteractions between a management component, server computing devices,and virtual machines instantiated on the server computing devices onbehalf of customers, one skilled in the art will appreciate that thetechniques disclosed herein may be applied to any number of hardware orsoftware processes or applications. For example, while virtual machineinstances will be generally used as the illustrative computing resourceconsumer, other programs or workloads may be substituted, such asapplication software, operating systems, storage area network (SAN)nodes, and the like. In addition, while computing resources such asmemory, CPU capacity, and network bandwidth will be used as theillustrative computing resources, other computing resources may besubstituted, such as network link traffic, latency, processorinstruction sets, and the like. Further, although various aspects of thedisclosure will be described with regard to illustrative examples andembodiments, one skilled in the art will appreciate that the disclosedembodiments and examples should not be construed as limiting. Variousaspects of the disclosure will now be described with regard to certainexamples and embodiments, which are intended to illustrate but not limitthe disclosure.

FIG. 1 illustrates an example network computing environment 100 in whichautomated profiling of resource usage and assignment of resources basedon those profiles may be implemented. Operating profiles and assignmentof resources can be based on prior measurements of actual resource usageand other operating metrics, and also on expected future usage ofresources. A network computing environment 100 can include a managementcomponent 102 and any number of physical host computing devices 104a-104 n in communication via a network 110. One or more customers 122may communicate with the components of the network computing environment100 via a network 120.

Network computing environments 100 such as the one illustrated in FIG. 1may be implemented in data centers and other environments in whichmultiple host computing devices 104 a-104 n provide computing servicesand resources to internal or external customers 122. As described inmore detail below, each customer 122 may connect to the managementcomponent 102 or some other component within the network computingenvironment 100 to initiate computing processes. The initiation ofcomputing processes may include instantiation of a virtual machineinstance on a host computing device 104 or the configuration of anoperating environment and one or more software applications. The virtualmachine instance may execute on behalf of the user, consuming computingresources of the host computing device 104, network 110, and the like.While the present disclosure will focus, for purposes of illustrationonly, on the operation of a network computing environment 100 providingcomputing services to external or internal customers 122 through the useof virtual machines, the systems and processes described herein mayapply to any implementation of a network computing environment 100,including one with no separate customer 122 entities or no virtualmachine usage.

Each host computing device 104 may be a server computer, such as a bladeserver. Optionally, a host computing device 104 may be a midrangecomputing device, a mainframe computer, a desktop computer, or any othercomputing device configured to provide computing services and resourcesto multiple consumers, such as virtual machine instances, concurrently.In a typical implementation, a host computing device 104 can beconfigured to communicate with other host computing devices 104, amanagement component 102, or some other component of the networkcomputing environment 100 via a network 110.

The network 110 may be a local area network (LAN), wide area network(WAN), some other network, or a combination thereof. In addition, thenetwork computing environment 100 may connect to another network 120,such as a corporate or university network, or a collection of networksoperated by independent entities, such as the Internet. Customers 122 ofthe network computing environment 100 may communicate with hostcomputing devices 104 over the combination of the networks 120, 110. Insome embodiments, the customers 122 may cause a computing device 102 tolaunch a virtual machine instance to execute various computingoperations for or on behalf of the customer 122. Any number of virtualmachine instances may be running on a single host computing device 104at a given time. In addition, the various virtual machine instancesrunning on a host computing device 104 may be associated with a singlecustomer 122 or with a number of different customers 122.

The management component 102 may be implemented as hardware or as acombination of hardware and software. For example, the managementcomponent 102 may be a computing system of one or more computing devicesconfigured to execute one or more software programs to perform thefunctions described herein. In some embodiments, the managementcomponent may include one or more of the host computing devices 104a-104 n.

FIG. 2 illustrates a sample management component 102 in greater detail.The management component 102 can include a profile determination module202, a placement module 204, a migration module 206, an operatingmetrics data store 208, and a profile data store 210. In someembodiments, the management component 102 may include more or fewermodules and data stores than those illustrated in FIG. 2. For example,there may be no separate migration module 206, when the migrationfeature is not implemented, or implemented by the placement module 204.In another example embodiment, there may be additional data stores forgeneralized customer profiles.

In operation, the profile determination module 202 can obtain operatingdata regarding operating metrics and resource usage by instances of aparticular virtual machine instance configuration at a particular time,of all virtual machine instances associated with a particular customer122, etc. The profile determination module 202 can analyze the operatingdata and develop an operating profile of the computing resourcesutilized by the virtual machine instance or group of virtual machineinstances being profiled. For example, operating data may includehistorical measurements regarding the amount of memory utilized, thecentral processing unit (CPU) utilization, the amount of network traffictransmitted or received, the amount of hard disk space utilized, thenumber of disk operations, the amount of electricity utilized (e.g.: theamount utilized by the host computing device 104 that may beattributable to the virtual machine instance), the amount of networklink traffic initiated, and the like. The profile determination module202 can then determine an average for each of the measurementsassociated with instances of a particular virtual machine instanceconfiguration or group of virtual machine instance configurations, andstore the averages in the operating profile. The operating profile neednot be limited to average measurements. For example, the operatingprofile may include other statistical analyses, such as the median,standard deviation, usage histogram or any other appropriate or usefuldata. In some embodiments, the operating profile may further becharacterized according to temporal characteristics of usage, such asthe time of day, day of the year, etc.

The operating profile may also be characterized according to expectedmeasurements and operating metrics. For example, a variance from anexpected performance metric, generally referred to as jitter, may beobserved and included in the operating profile. Such data may be used todetermine whether design goals, service-level agreements and otherpromises or obligations to the consumer are being met or to determinehow often they fail to be met. The placement module 204 may account forjitter when making future placement decisions, endeavoring to ensurethat the same operating metric will not fall outside the expected rangeor otherwise ensuring that consumer obligations are satisfied. In someembodiments, the operating profile may contain other data, such aslatency preferences or requirements, instructions set preferences orrequirements, and the like. Such data may be provided by consumers ordetermined through analysis of virtual machine instance operation by theprofile determination module 202.

Illustratively, a service provider may provide three classes of virtualmachines: small, medium, and large. Each class may be associated with apredetermined amount of each computing resource that will be reservedfor use by instances of the virtual machine (e.g.: small VMs may have 2GB RAM, medium VMs may have 8 GB RAM, large VMs may have 32 GB RAM).Customers may instantiate instances of a virtual machine configured withan operating system and application software, such as a large virtualmachine configured with web server software. Measurements may berecorded regarding usage of computing resources by an instance of thelarge virtual machine configured with web server software. The profiledetermination module 202 can then calculate expected resource usageamounts for future instances of the virtual machine instanceconfiguration when, for example, used as a web server. The expectedresource usage amounts may form the basis of the operating profiledetermined by the profile determination module 202. The profiledetermination module 202 may then modify the operating profile as a dataset including measurements of actual resource usage is built over time.

The profile data that is used by the profile determination module 202may be obtained from a variety of sources. As described above, the datamay be obtained from an entity associated with the virtual machine. Datamay also be obtained directly from a workload analysis component of thehost computing device 104 on which the virtual machine instance isexecuting. In some embodiments, the data can be obtained from anoperating metric data store 208. The operating metric data store 208 maybe integrated with the management component 102, as illustrated in FIG.2, or it may be physically located on a separate computing device, suchas a dedicated relational database management system (RDBMS) server. Theoperating profiles that are determined by the profile determinationmodule 202 may be stored in a profile data store 210. Similar to theoperating metric data store 208, the profile data store 210 may beintegrated with the management component 102 or located on separatecomputing device, such as a dedicated RDBMS server.

In some network computing environments 100, there may be thousands ormore of virtual machine instances to profile, and each operating profilemay, for example, be based on the analysis of usage data unique toparticular virtual machine instance configurations or the usage of aparticular customer. In order to efficiently utilize the operatingprofiles to make placement decisions regarding the instantiation ofvirtual machine instances on host computing devices 104 a-104 n, theoperating profiles may be generalized. Accordingly, a number ofdifferent virtual machine instance configurations may be associated withthe same, or substantially similar, operating profiles even though theremay be variances in the actual resource usage associated with eachvirtual machine instance configuration. For example, the profiledetermination module 202 may associate a virtual machine instanceconfiguration with predefined expected usage amounts rather than storinga customized operating profile for each virtual machine instanceconfiguration. The predefined operating profile may include utilizationranges for each computing resource that is measured. In addition, theoperating profiles may be hierarchical, such that a particular virtualmachine instance configuration is associated with a particular operatingprofile, and also with a more general operating profile that is itselfassociated with multiple virtual machine instance configurations.

In some embodiments, the operating profiles may be further generalizedinto categories. For example, a number of virtual machine instanceconfigurations, each associated with a different amount of networkusage, may be categorized as “light network applications” or “heavynetwork applications” depending on whether the usage measurement exceedsor falls short of some threshold. In such a categorization scheme, avirtual machine instance configuration that, when instantiated,primarily performs local computing operations and rarely utilizes anetwork connection may be categorized in the same “light networkapplications” category as a virtual machine instance configuration thatoften utilizes a network connection, but only for very smalltransmissions which may be trivial in comparison to the amount ofnetwork bandwidth available to the host computing devices 104 a-104 n onwhich the virtual machine instance executes. Such generalized operatingprofiles may also be based on a composite of two or more categories,such as “light network application/heavy CPU application” and “lightnetwork application/light CPU application.” Returning to the previousexample, the two virtual machine instance configurations may beassociated with different categories. The virtual machine instanceconfiguration that, when instantiated, primarily performs localcomputing operations and rarely utilizes a network connection may becategorized as a “light network application/heavy CPU application,”while the virtual machine instance configuration which, wheninstantiated, often initiates small network transmissions may becategorized as a “light network application/light CPU application” ifthe CPU utilization of the virtual machine instances fall below athreshold.

As described above, the operating profiles, whether specific to avirtual machine instance configuration or generalized to a number ofvirtual machine instance configurations, may be used to identify a hostcomputing device 104 a-104 n on which to place virtual machineinstances. The placement module 204 may be invoked when a customer 122initiates a computing session or when a virtual machine is otherwiseinstantiated. The placement module 204 may determine which operatingprofile is associated with the virtual machine instance at the currenttime. For example, the operating profile may be a customized profileincluding measurements of actual resource usage associated with thevirtual machine instance at the current time of day, during the currentmonth of the year, etc. In some cases, the measurements may be specificto a particular customer, such that an operating profile for aparticular customer may be created and accessed. The customer-specificoperating profile can apply to a specific virtual machine instanceconfiguration or it may generally apply to multiple distinct virtualmachine instance configurations. Optionally, the operating profile maybe a generalized profile based on the overall character of resourceusage associated with the virtual machine instance, which may also bebased on the current time of day, etc. The virtual machine placementmodule 204 can then select a host computing device 104 on which tolaunch the virtual machine instance based on the resource availabilityof the host computing devices 104 a-104 n and the expected resourceusage of the virtual machine instance determined from the operatingprofile.

Resource utilization may be dynamic over the lifetime of a singleinstance of a specific virtual machine instance configuration, and overmultiple instances of the specific virtual machine instanceconfiguration. The migration module 206 of the management component 102may monitor the resource utilization of each executing virtual machineinstance and the host computing device 104 on which the virtual machineinstance is executing. When the resource utilization changes, themigration module 206, similar to the virtual machine placement module204 described above, may select an appropriate host computing device 104on which to place the virtual machine instance. A new instance of thevirtual machine may be launched on the selected host computing device104, and the execution state of the virtual machine instance (memory,inputs, and the like) may be copied to the new virtual machine instance.When the new virtual machine instance is ready to begin executing, theprevious virtual machine instance may be terminated without a loss ofdata and without a substantial loss of performance. The new virtualmachine instance may execute more efficiently due to the availableresources.

In some embodiments, rather than instantiating a new instance of thevirtual machine on a different host computing device and terminating theprevious instance, resources may be reallocated. When resourceutilization or performance metrics change, additional resources (e.g.,memory) may be allocated to the particular virtual machine. For example,a resource may be reallocated from other virtual machines that are notexpected to fully utilize the resource.

Turning now to FIG. 3, an illustrative process 300 for determining anoperating profile for a virtual machine instance configuration will bedescribed. The process 300 may be executed by a management component102. The management component 102 may receive a request from a customer122 or otherwise be notified to instantiate a virtual machine. Afterinstantiating the virtual machine instance, identifying an instantiatedvirtual machine instance or causing the virtual machine to beinstantiated, the management component 102 may monitor or otherwisereceive operating data regarding computing resource utilizationassociated with the virtual machine instance. Based on the resourceusage and operating metric data, the management component 102 candetermine or update an operating profile for the virtual machineinstance configuration, or update an existing operating profile.Advantageously, the operating profile may be compared with otheroperating profiles and generalized and the virtual machine instanceconfiguration may be associated with a category of resource usage.

The process 300 begins at block 302. The process 300 may beginautomatically, such as in response to the receipt of a request toinstantiate a virtual machine. For example, the process 300 may beembodied in a set of executable program instructions and stored on acomputer-readable medium drive of the computing system with which themanagement component 102 is associated. When the process 300 isinitiated, the executable program instructions can be loaded intomemory, such as RAM, and executed by one or more processors of thecomputing system. In some embodiments, the computing system may includemultiple computing devices, such as servers, and the process 300 may beexecuted by multiple servers, serially or in parallel.

At block 304, the management component 102 or some other componentlaunches a virtual machine instance. As described in detail below withrespect to FIG. 5, the management component may select a host computingdevice 104 on which to launch the virtual machine instance based on theresources expected to be consumed by the virtual machine instance andthe resources that the host computing devices 104 a-104 n currently haveavailable. The resources expected to be consumed by the virtual machineinstance or to be made available to the virtual machine instance may bedetermined from a preexisting operating profile, from informationreceived from the customer 122 or other entity requesting that thevirtual machine instance be launched, etc.

The process 300 may proceed to block 306 for the newly launched virtualmachine instance in order to obtain operating metrics and to create ormodify an operating profile. While the process 300 proceeds, any numberof additional virtual machine instances may be launched and/or placed atblock 304 based on the same operating profile, either as it originallyexisted, or as modified during the execution of the process 300 forpreviously launched virtual machine instances. In this way, the process300 may be performed in any number of concurrent instances, generallycorresponding to the number of virtual machine instances associated withthe operating profile (or, in a hierarchy of profiles, a profile from ahigher level in the hierarchy) that may be executing at a particulartime.

At block 306, the resources utilized by the virtual machine instance maybe monitored, and resource usage measurements and other operatingmetrics may be obtained. At block 308, the operating metrics may berecorded. The monitoring may be performed by the management component102, or by some other component, such as a workload analysis component421 of the host computing device 104 on which the virtual machineinstance is executing. The operating metrics may be stored at theoperating metrics data store 208. In embodiments using a workloadanalysis component 421, the workload analysis component 421 may storeoperating metrics temporarily or long-term. The workload analysiscomponent may transmit data regarding the operating metrics to themanagement component 102 for storage in substantially real time, atscheduled intervals, upon virtual machine termination, at some othertime, or not at all.

FIG. 4 illustrates measurement of the utilization of several resourcesprovided to multiple virtual machine instances by a host computingdevice 104. As illustrated in FIG. 4, a host computing device 104 mayprovide computing resources, such as memory 402, a CPU 404, and anetwork bandwidth 406. In some embodiments, additional or fewercomputing resources may be provided to virtual machine instances. Forexample, a virtual machine instance may not be permitted to communicatewith other devices, and therefore utilization of the network interface406 need not be measured. In another example, a host computing device104 may provide and track utilization of hard disk space, hard diskoperations, electrical power, and the like.

In some embodiments, a provider of computing resources, such as anoperator of a network computing environment 100, may provide customerswith a set amount of computing resources on which to execute a virtualmachine instance. For example, a customer 122 may reserve for one of itsvirtual machine instance configurations a predetermined amount ofmemory, such as random access memory (RAM), a predetermined amount ofcomputing capacity, such as CPU cores, and a predetermined amount ofnetwork bandwidth, as provided by a network interface. Memory 402 of ahost computing device 104 may be segregated into portions 410, 412, 414which are reserved for single virtual machine instances (e.g.: portions412, 414) or for the operation of the host computing device 104 andother internal procedures (e.g.: portion 410). The portion reserved foroperation of the host computing device 104 may include a hypervisor forassisting in the launch, execution, and termination of virtual machineinstances, an operating system, drivers, and the like. In addition, thehost computing device 104 may include a workload analysis component 421which monitors resource utilization and optionally communicates with themanagement component 102. The workload analysis component 421 may alsoreside in the memory space 410, and may be integrated into thehypervisor 420 or may be an independent component which shares thememory space 410. In some embodiments, the workload analysis component421 may reside in a memory space 412, 414 reserved for customer virtualmachine instances. In such cases, the workload analysis component 421may be integrated into the virtual machine instance configurations orincluded in the virtual machine instance upon instantiation. In furtherembodiments, the workload analysis component 421 may reside in aseparate memory space reserved for it, or may be implemented as acomponent, such an independent hardware device, which does not share thememory 402 of the host computing device 104.

In many cases, a virtual machine instance may not utilize the entireportion of a resource that is reserved for it. For example, VM1 422,illustrated in FIG. 4, may be a virtual machine instance of a customer122, and may be launched into memory space 412, the entirety of which isreserved for use by VM1 422. In operation, VM1 422 may not utilize theentire memory space 412, and in some cases may utilize on a smallfraction of the reserved memory space 412. At times, however, theutilization of the memory space 412 may change, and VM1 422 may utilizesubstantially all of the memory space 412. The workload analysiscomponent 421 may monitor these changes and record measurements andother data, such as the time of day, the specific virtual machineinstance configuration, or which other virtual machine instances, ifany, were executing on the host computing device 104. The workloadanalysis component 421 may transmit the data to the management component102 or to a data store. In some embodiments, the workload analysiscomponent 421 may temporarily store the data and later transfer it tothe management component 102, such as on a schedule, or in response to atriggering event, such as the termination of VM1 422. Similar tomeasuring and recording data about the utilization of memory 402, theworkload analysis component 421 or some other component may monitorusage of the CPU 404, network interface 406, or any other computingresource utilized by VM1 422.

Data may be obtained and recorded regarding any variances from expectedor preferred operating metrics. For example, resource usage measurementsand other operating metrics may be recorded and compared to theoperating profile in order to determine whether there is a variance froman expected or preferred metric. In some cases, the operating metricsmay be recorded on a customer-by-customer basis. Data regarding avariance may be recorded so that future placement or migration decisionsmay be made based on the variance. In addition, data regarding off-hostresources, such as latency, link traffic, and the like may be recorded.The workload analysis component 421 may record such data, or somecomponent external to the host (e.g., the management component 102 or aswitch) may observe the operating metrics. In some embodiments, resourceusage that does not necessarily reduce the availability of the resourcemay be determined. For example, if a virtual machine instance orapplication software running thereon performs certain cryptographicoperations or is observed calling certain cryptographic functions orinstructions, such data may be recorded. The placement module 204 ormigration module 206 may consider such data when launching or migratingan instance of the virtual machine. A host computing device may beselected which provides more efficient or more powerful cryptographicinstructions, such a device supporting Intel® Advanced EncryptionStandard (AES) New Instructions (AES-NI) or similar device.

At block 310, the profile determination module 202 or some other moduleof the management component 102 may modify an operating profileassociated with the virtual machine instance, or create a new operatingprofile. As described above, operating profiles may include informationabout typical or expected resource usage, variances from expected ordesired operating metrics, and the like. For example, the operatingprofile may consist of average measurements for each of a number ofdifferent instances of a single virtual machine instance configuration.Each resource may be associated with multiple measurements whichcorrespond to operating based on a particular customer, a time of day, aday of the year, or other environmental factors.

In some embodiments, each resource of the operating profile may beassociated with a score or some other indication of utilization ratherthan a statistical measurement. For example, each resource may beassigned a score of 1-10, where higher numbers are associated with theheaviest and/or most frequent users of a resource. In some embodiments,the operating profiles may be generalized further. A predefined set ofgeneralized operating profiles may cover ranges of measurements orscores for each resource. For example, the virtual machine instanceconfiguration from which VM1 422 was instantiated may be assigned to onegeneralized operating profile if, during nighttime hours, VM1 422utilizes no more than 25% of its memory space 412 but utilizes almost100% of its CPU availability. The generalized operating profiles mayinclude multiple ranges of measurements for each resource, depending onthe time of day or other factors. Returning the previous example, thevirtual machine instance configuration from which VM1 422 isinstantiated may be instead assigned to a different predefined operatingprofile if the virtual machine instances typically utilize resources inthat manner described above during nighttime hours, but during daytimehours it utilizes 50% of both its memory segment 412 and CPUavailability. Multiple generalized operating profiles may be assigned toparticular virtual machine instance configurations based on usage byparticular customers. For example, each customer that users the virtualmachine instance configuration may be associated with a differentoperating profile.

In some embodiments, each customer 122 may be associated with ageneralized operating profile even though it has a number of differentvirtual machine instance configurations, and even though each virtualmachine instance configuration may utilize resources differently. Acustomer 122 may have one virtual machine instance configuration, suchas the one from which VM1 422 in FIG. 4 is instantiated, which may beindependently profiled as a light CPU application, while another virtualmachine instance configuration, such as the one from which VM2 424 isinstantiated, may be independently profiled as a heavy CPU application.The customer 122 may be profiled as a moderate CPU user, because itsaverage CPU use is moderate. Optionally, the customer 122 may beprofiled as a heavy CPU user, because it has at least one virtualmachine instance configuration which is a heavy CPU application. Inother embodiments, customers may have several associated operatingprofiles for each virtual machine image configuration. Differentcustomers or users of substantially the same virtual machine image, suchas VM1 422, may use different amounts of resources, even though thevirtual machine image is a common configuration. A given customerstarting a particular VM may be more likely to use that VM in the sameway as previously recorded, and consume approximately the sameresources.

Operating profiles for each virtual machine instance configuration maybe stored in the profiles data store 210. The actual measurements foreach profile may be stored in the operating profile, or an ID or otherindication of which category or generalized operating profile thevirtual machine instance configuration is associated with may be stored.In embodiments which determine and utilize customer profiles instead ofor in addition to virtual machine profiles, customer profile data may bestored in the same data store 210 or in a different data store.

At block 312, related or generalized operating profiles may be createdor modified. For example, higher-level profiles may be created ormodified if hierarchical profiles are used. Historical operating metricsmay be accessed from the operating metrics data store 208 for eachvirtual machine instance configuration associated with the high-leveloperating profile, in some cases regardless of which lower-levelprofiles the virtual machine instance configurations are associatedwith. Statistical analyses may be performed and operating metricvariances may be determined as described above. Advantageously, themodified high-level or generalized operating profile may be accessed andused by the placement module 204 or migration module 206 wheninstantiating or migrating virtual machine instances that are differentfrom the virtual machine instance associated with the current executionof the process 300. Accordingly, the recorded operating metricsassociated with one virtual machine instance may be used to fine tunethe placement and execution of other virtual machine instances, eventhose which are not instantiated from the same virtual machine instanceconfiguration.

Turning now to FIG. 5, an illustrative process 500 for determiningplacement of virtual machine instances based on operating profiles willbe described. The process 500 may be executed by a management component102. The management component 102 may receive a request from a customer122 or may otherwise be notified to launch an instance of a virtualmachine from a specific virtual machine instance configuration or image.The management component 102 can identify host computing devices 104a-104 n which are able to host the virtual machine instance anddetermine the current status of the host computing devices 104 a-104 nwith respect to available computing resources. Advantageously, themanagement component 102 may also obtain an operating profile for thevirtual machine instance configuration to be instantiated, and determinewhich of the available host computing devices 104 a-104 n may mostefficiently host the virtual machine from the standpoint of availableresources. A host computing device 104 may be selected which is alreadyexecuting virtual machines and which has committed most or all of itsresources to host virtual machines already executing. Based on operatingprofiles of the currently executing virtual machine instances and of thevirtual machine instance to be launched, the management component 102may launch the virtual machine instance on the host computing device 104if the management component 102 determines that the host computingdevice 104 can provide the computing resources that the virtual machineinstances will likely consume. In some cases, this may includeoversubscribing resources (e.g., allocating the same resources tomultiple virtual machine instances). Moreover, the management component102 can monitor the execution of the virtual machine instances on thehost computing device 104, and transfer execution of one or more virtualmachine instances to another host computing device 104 if there are notenough computing resources to satisfy each virtual machine instance.

The process 500 begins at block 502. The process 500 may be initiatedautomatically, such as in response to the receipt of a request to launcha virtual machine instance. For example, the process 500 may be embodiedin a set of executable program instructions and stored on anon-transitory computer-readable medium drive of the computing systemwith which the management component 102 is associated. When the process500 is initiated, the executable program instructions can be loaded intomemory, such as RAM, and executed by one or more processors of thecomputing system. In some embodiments, the computing system may includemultiple computing devices, such as servers, and the process 500 may beexecuted by multiple servers, serially or in parallel.

At block 504, the management component 102 may receive a request or someother notification to initialize a virtual machine instance. Thenotification may be received from a customer 122, a host computingdevice 104, or some other component or entity. In some embodiments, avirtual machine instance may request initialization of another virtualmachine instance, another instance of the same virtual machine instanceconfiguration or image, etc.

At block 506, the VM placement module 204 or some other module of themanagement component 102 may obtain an operating profile for the virtualmachine instance to be launched. The operating profile may be loadedfrom the profile data store 210 or obtained from some other source. TheVM placement module 204 may inspect the operating profile to determinewhich resources the virtual machine instance is likely to utilize and inwhich quantity. As described above, the operating profile of the virtualmachine instance configuration may be different depending onenvironmental factors, such as the time of day. In such cases, the VMplacement module 204 of the management component 102 can consider suchenvironmental factors when inspecting the operating profile.

At block 508, the VM placement module 204 or some other module of themanagement component 102 may select a host computing device 104 on whichto launch the virtual machine instance based on available resources andthe operating profile. For example, a network computing environment 100may include a number of host computing devices 104 a-104 n. The hostcomputing devices 104 a-104 n need not be identical; some may have moreor less RAM than others, more or less powerful processors or a differentnumber of processors, etc. The VM placement module 204 may select a hostcomputing device 104 on which to launch the virtual machine instancebased on the expected resource utilization as identified by theoperating profile and by the resources that each computing device makesavailable.

In some embodiments, a host computing device 104 may be configured tohost a set number of instances of a particular virtual machine or classof virtual machines. As shown in FIG. 4, the host computing device 104may have an amount of memory 402 such that it can reserve apredetermined memory space 410 for the hypervisor 420, and twoadditional memory spaces 412, 414 of a predetermined size for virtualmachines. Two virtual machine instances 422, 424 may be launched on thehost computing device 104, with each virtual machine instance 422, 424assigned a separate memory space 412, 414. A customer 122 may reserve aparticular amount of a resource to be available to its virtual machineinstances 422, 424, such as by selecting a particular class of virtualmachine (e.g.: small, medium or large as described above) to configure.The memory spaces 412, 414 may correspond to the maximum allowableamount of resources available to the virtual machine instances 422, 424,as reserved by the customer 122. However, in practice the virtualmachine instances 422, 424 may not utilize the entire amount of acomputing resource that is reserved for them. For example, as seen inFIG. 4, the virtual machine instances 422, 424 are only utilizing afraction of the memory spaces 412, 414 that are reserved for them. Itmay be advantageous to utilize such excess memory space and other excesscomputing resources so as to reduce the number of host computing devices104 a-104 n required to service all currently executing virtual machineinstances or to more efficiently utilize the resources of those hostcomputing devices 104 a-104 n which are operating.

FIG. 6 illustrates a host computing device 104 with oversubscribedcomputing resources. A third virtual machine instance 426 has beenlaunched on the host computing device 104 even though the host computingdevice 104 only contains two memory spaces 412, 414 available forvirtual machine instances. Based on the operating profile associatedwith each of the virtual machine instances 422, 424, 426, the VMplacement module 204 may determine that VM2 422 utilizes only a fractionof its available memory space 414, and VM3 426 also uses only a fractionof its available memory space when it is launched. Therefore, the VMplacement module 204 may launch VM3 426 on the same computing device asVM2 424 and assign them to the same memory space 414.

As seen in FIG. 6, VM1 422 utilizes substantially all of its memoryspace 412, and therefore the VM placement module 204 may not assignanother virtual machine instance to the same memory space 412 due to theoperating profile of VM1 422. However, VM1 422 utilizes only a smallamount of CPU capacity 442, and therefore a host computing device 104 onwhich VM1 422 is executing may be a candidate for oversubscription ifthe operating profiles of the virtual machine instances arecomplementary. In the example illustrated in FIG. 6, VM2 424 utilizes alarge amount of CPU capacity 444. However, if the operating profile ofVM3 426 indicates that it is a light user of CPU capacity, then thethree virtual machine instances VM1 422, VM2 424, and VM3 426 may becandidates for oversubscription due to the complementary, rather thanoverlapping, operating profiles of the virtual machine instanceconfigurations from which they are instantiated. Additional resourcesmay be factored into an oversubscription determination in order toensure that each virtual machine instance executing on a host computingdevice 104 has readily available to it the amount of each computingresource that it typically requires. For example, network bandwidthutilization 406, as illustrated in FIG. 6, also supports theoversubscription determination example described above because, eventhough the operating profile for VM3 426 may indicate that it is a heavynetwork bandwidth application, VM1 422 and VM2 424 utilize only a smallamount of network bandwidth.

At block 510, the VM placement module 204 or some other module of themanagement component 102 may place the virtual machine instance on thehost computing device 104 that is identified in block 508. As describedabove, the virtual machine instance may be placed on a host computingdevice 104 with other virtual machine instances which have reservedamounts of computing resources totaling or exceeding the amount providedby the host computing device 104. In some cases, the oversubscriptionmay be substantial.

FIG. 7 illustrates a substantially oversubscribed host computing device104. As described above, customers 122 may reserve a specified amount ofcomputing resources for use by virtual machine instances of thecustomer. However, in some cases a customer 122 may have substantiallyoverestimated the amount of computing resources that its virtual machineinstances may actually use. In extreme cases, customer 122 may reserve alarge quantity of computing resources, launch virtual machine instancesassociated with those resources, and then let the virtual machineinstances sit idle or otherwise substantially underuse the reservedresources. Over the course of time, operating profiles may be developedfor the virtual machine instances or for the customer 122 which reflectthe substantial underuse of reserved resources. The management component102 may then launch a large number of such low-utilization virtualmachine instances on a single host computing device 104. The hostcomputing device 104 of FIG. 7 includes seven different virtual machineinstances 422, 424, 426. 722, 724, 726, 728 sharing computing resourceswhich may typically be reserved for only two virtual machine instances,as described above with respect to FIGS. 4 and 6. However, there isstill excess capacity of each of the three computing resources 402, 404,406. In some cases, hundreds or more of virtual machine instances may beplaced on a host computing device 104, such as when the host computingdevice 104 has a large amount of available computing resources and thevirtual machine instances are substantially idle.

At block 512, the resource utilization of each virtual machine instancemay be monitored. Over the lifetime of the specific virtual machineinstances, the workload analysis component 421 may monitor operating andnotify the management component 102 if one of the virtual machineinstances begins to utilize resources at a level that is not serviceableby an oversubscribed host computing device 104, or if the resource usageor an operating metric otherwise differs from an expected or desiredamount. In some embodiments, the management component 102 may performthe monitoring.

At block 514, the management component 102 can determine whetherresource usage or an operating metric differs from an expected ordesired amount. For example, the management component can determinewhether a change in resource usage exceeds a threshold or may otherwisecause undesirable performance degradation. A virtual machine instancewhich begins to utilize more of a computing resource than expected,based on its operating profile and the placement determined by themanagement component 102, may be transferred to a host computing device104 that is oversubscribed to a lesser extent, or to a host computingdevice 104 that is not oversubscribed at all. In such cases, executionof the process 500 can return to block 508, where the VM migrationmodule 206 or some other management component 102 determines to whichcomputing device to transfer the virtual machine 842.

FIG. 8 illustrates a host computing device 104 a which hasoversubscribed CPU capacity 404 a. A virtual machine instance VM4 842may begin to consume a large amount of available CPU capacity 404 a, incontradiction to its operating profile. However, the customer 122associated with the virtual machine instance VM4 842 may have reserved alarge amount of CPU capacity for the virtual machine instanceconfiguration from which VM4 842 is instantiated, and therefore it maybe desirable to provide the virtual machine instance VM4 842 with moreCPU capacity than an oversubscribed host computing device 104 a canprovide. As shown in FIG. 8, the host computing device 104 b may be acandidate for such a transfer. The virtual machine instance VM8 844 iscurrently consuming only a small fraction of the CPU capacity 404 bavailable on the host computing device 104 b, and the managementcomponent 102 may determine that the operating profile associated withVM8 844 indicates that it is not likely to consume more. The VMmigration module 206 can initiate transfer of VM4 842 from hostcomputing device 104 a to host computing device 104 b.

Transfer of a virtual machine instance may include first launching aninstance of the same virtual machine instance configuration or image onthe target host computing device 104 b while the virtual machineinstance on the source host computing device 104 a continues to execute.The execution state of the virtual machine instance on the source hostcomputing device 104 b, including the data in the memory space or harddisk associated with the virtual machine instance, network connectionsestablished by the virtual machine instance, and the like, can then beduplicated at the target host computing device 104 b. The virtualmachine instance on the source host computing device 104 a can beterminated, and the virtual machine instance on the target hostcomputing device 104 b can continue execution from that point.

In some embodiments, the initial placement or transfer of a softwareworkload (e.g., an application or storage node) may be associated withvirtual machine instance use of, or interaction with, the softwareworkload, even though the software workload may not necessarily be avirtual machine instance itself. For example, a software workload mayconsist of a storage node implemented as an agent storing data accordingto the expectations of virtual machine instances that may be using thedata. In such cases, initial placement or transfer of the workload (inthis case, the storage node) may be based on placing it near orotherwise making it more accessible by the virtual machine instance orinstances that will be using it.

Depending on the embodiment, certain acts, events, or functions of anyof the processes or algorithms described herein can be performed in adifferent sequence, can be added, merged, or left out altogether (e.g.,not all described operations or events are necessary for the practice ofthe algorithm). Moreover, in certain embodiments, operations or eventscan be performed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors or processor cores or onother parallel architectures, rather than sequentially.

The various illustrative logical blocks, modules, routines, andalgorithm steps described in connection with the embodiments disclosedherein can be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,and steps have been described above generally in terms of theirfunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. The described functionality can beimplemented in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the disclosure.

The steps of a method, process, routine, or algorithm described inconnection with the embodiments disclosed herein can be embodieddirectly in hardware, in a software module executed by a processor, orin a combination of the two. A software module can reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of a non-transitorycomputer-readable storage medium. An exemplary storage medium can becoupled to the processor such that the processor can read informationfrom, and write information to, the storage medium. In the alternative,the storage medium can be integral to the processor. The processor andthe storage medium can reside in an ASIC. The ASIC can reside in a userterminal. In the alternative, the processor and the storage medium canreside as discrete components in a user terminal.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list.

Conjunctive language such as the phrase “at least one of X, Y and Z,”unless specifically stated otherwise, is to be understood with thecontext as used in general to convey that an item, term, etc. may beeither X, Y, or Z, or a combination thereof. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of X, at least one of Y and at least one of Z toeach be present.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it can beunderstood that various omissions, substitutions, and changes in theform and details of the devices or algorithms illustrated can be madewithout departing from the spirit of the disclosure. As can berecognized, certain embodiments of the inventions described herein canbe embodied within a form that does not provide all of the features andbenefits set forth herein, as some features can be used or practicedseparately from others. The scope of certain inventions disclosed hereinis indicated by the appended claims rather than by the foregoingdescription. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

What is claimed is:
 1. A system for profiling computing resource usage,the system comprising: one or more processors; a computer-readablememory; and a management module comprising executable instructionsstored in the computer-readable memory, the management module, whenexecuted by the one or more processors, configured to: receive a requestfor initialization of a new instance of a virtual machine instanceconfiguration, the virtual machine instance configuration associatedwith an expected resource usage amount of a first computing resource,wherein the expected usage amount is based at least in part on aplurality of prior measurements associated with usage, by instances ofone or more a similar virtual machine instance configurations, of thefirst computing resource; and in response to the request; identify acomputing device of a plurality of computing devices based at least onwhether an available amount of the first computing resource on thecomputing device is greater than the expected amount; and cause, atleast in part, the new instance to be initialized on the computingdevice.
 2. The system of claim 1, wherein usage of the first computingresource comprises one of central processing unit (CPU) utilization,memory utilization, network utilization, hard disk utilization, orelectrical power utilization.
 3. The system of claim 1, wherein themanagement module, when executed, is further configured to: obtain theplurality of prior measurements regarding usage of the first computingresource; and determine an operating profile for the virtual machineinstance configuration based at least in part on the plurality of priormeasurements, wherein the operating profile comprises the expectedresource usage amount.
 4. The system of claim 3, wherein the operatingprofile further comprises a desired operating characteristic of thecomputing device.
 5. The system of claim 4, wherein the desiredoperating characteristic relates to memory capacity, central processingunit (CPU) capacity, network bandwidth, network latency, position withina network topology, instruction set, or variance of a performancemetric.
 6. The system of claim 4, wherein identifying the computingdevice comprises determining that a characteristic associated with thecomputing device corresponds to the desired characteristic.
 7. A systemfor profiling computing resource usage, the system comprising: one ormore processors; a computer-readable memory including executableinstructions that, when executed by the one or more processors,configure the system to: determine an operating constraint for aninstance of a virtual machine based at least in part on operatingmetrics determined from running at least an instance of a similarvirtual machine; receive a request to instantiate the virtual machine;and in response to the request: identify a host computing device, of aplurality of host computing devices, associated with one or moreoperating characteristics related to the operation of virtual machineinstances based partly on whether the one or more characteristicssatisfy the operating constraint; and cause at least in part, a newinstance of the virtual machine to be instantiated on the host computingdevice.
 8. The system of claim 7, wherein at least one of the operatingmetrics relates to central processing unit (CPU) utilization, memoryutilization, network utilization, hard disk utilization, or electricalpower utilization.
 9. The system of claim 7, wherein at least one of theone or more characteristics comprises memory capacity, centralprocessing unit (CPU) capacity, network bandwidth, network latency,position within a network topology, instruction set, or variance of aperformance metric.
 10. The system of claim 7, wherein the operatingconstraint relates to an expected usage amount of a computing resourceprovided by the host computing device.
 11. The system of claim 10,wherein the expected usage amount is further based at least in part ondata received from a customer associated with the virtual machine. 12.The system of claim 7, wherein the module, when executed, is furtherconfigured to: receive an additional operating metric regardingoperation of the new instance on the host computing device, theadditional operating metric related to the one or more operatingcharacteristics; and in response to determining, based on the additionaloperating metric, that the one or more operating characteristics nolonger satisfy the operating constraint, transfer the new instance to asecond computing device associated with one or more additional operatingcharacteristics that satisfy the operating constraint.
 13. Acomputer-implemented method for profiling computing resource usage, thecomputer-implemented method comprising: receiving, by a data centermanagement component comprising one or more computing devices, a requestfor initialization of a software workload associated with an operatingprofile, wherein the operating profile is based at least in part on aplurality of historical operating metrics associated with the softwareworkload; and in response to the request: identifying a computing deviceof a plurality of computing devices based at least in part on theoperating profile and one or more operating characteristics associatedwith the computing device; and causing the software workload to beinitialized on the computing device.
 14. The computer-implemented methodof claim 13, wherein the software workload comprises a virtual machineinstance, an operating system, a storage area network (SAN) node, or anapplication.
 15. The computer-implemented method of claim 13, wherein atleast one of the plurality of historical operating metrics relates tocentral processing unit (CPU) utilization, memory utilization, networkutilization, hard disk utilization, or power utilization.
 16. Thecomputer-implemented method of claim 13 wherein at least one of the oneor more operating characteristics comprises memory capacity, centralprocessing unit (CPU) capacity, network bandwidth, network latency,position within a network topology, instruction set, or variance of aperformance metric.
 17. The computer-implemented method of claim 13,wherein the operating profile comprises a first expected resource usageamount associated with a first computing resource, wherein the firstexpected resource usage amount is based at least in part on a pluralityof historical operating metrics regarding usage of the first resource,and wherein a characteristic of the one or more characteristic comprisesavailability of the first computing resource.
 18. Thecomputer-implemented method of claim 17, wherein each of the pluralityof historical operating metrics regarding usage of the first resource isassociated with a time that a measurement of usage was recorded, andwherein the operating profile is further based at least in part on thetime that each of the plurality of historical operating metrics wasrecorded.
 19. The computer-implemented method of claim 13, furthercomprising obtaining at least a portion of the plurality of historicaloperating metrics from an operation analysis component associated witheither the software workload or a computing device of the plurality ofcomputing devices executing the software workload.
 20. Thecomputer-implemented method of claim 13, further comprising determiningthe operating profile based at least in part on the plurality ofhistorical operating metrics.
 21. The computer-implemented method ofclaim 13, further comprising determining the operating profile based atleast in part on a service level agreement with a customer associatedwith the software workload.
 22. The computer-implemented method of claim13, wherein the operating profile comprises a desired operatingcharacteristic of the computing device.
 23. The computer-implementedmethod of claim 22, wherein identifying the computing device is furtherbased at least in part on determining that an operating characteristicof the one or more operating characteristics associated with thecomputing device corresponds to the desired operating characteristic.24. The computer-implemented method of claim 22, further comprising:receiving a substantially current operating metric regarding operationof the software workload on the computing device, the substantiallycurrent operating metric related to the one or more operatingcharacteristics; and in response to determining, based on thesubstantially current operating metric, that none of the one or moreoperating characteristics associated with the computing devicecorrespond to the desired operating characteristic, transferring thesoftware workload to a second computing device associated with anoperating characteristic corresponding to the desired operatingcharacteristic.
 25. The computer-implemented method of claim 13, whereinthe operating profile comprises a first predefined operating profile ofa plurality of predefined operating profiles.
 26. Thecomputer-implemented method of claim 25, wherein the predefinedoperating profiles are associated with levels in an operating hierarchy,and wherein the first predefined operating profile is associated withhigher level, in relation to the software workload, of the operatinghierarchy.
 27. The computer-implemented method of claim 13, wherein theoperating profile comprises a customer-specific operating profile, andwherein the plurality of historical operating metrics are associatedwith initialization or use of the software workload by the customer. 28.The computer-implemented method of claim 13, wherein the operatingprofile comprises a median, standard deviation, or usage histogram of ahistorical operating metric.
 29. The computer-implemented method ofclaim 13, wherein the operating profile is based at least in part onhistorical operating metrics from a particular time period.